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THE ENVIRONMENT OF MASSIVE QUIESCENT COMPACT GALAXIES AT 0.1 < Z < 0.4 IN THE COSMOS 

FIELD 

IvANA DamjanovS H. Jabran Zahid^, Margaret J. Geller^, Ho Seong Hwang® 

ABSTRACT 

We use Hectospec mounted on the 6.5-meter MMT to carry out a redshift survey of red {r — i> 0.2, 
g — r > 0.8, r < 21.3) galaxies in the COSMOS field to measure the environments of massive compact 
quiescent galaxies at intermediate redshift. The > 90% complete magnitude limited survey includes 
redshifts for 1766 red galaxies with r < 20.8 covering the central square degree of the field; 65% of 
the redshifts in this sample are new. We select a complete magnitude limited quiescent sample based 
on the rest-frame UV J colors. When the density distribution is sampled on a scale of 2 Mpc massive 
compact galaxies inhabit systematically denser regions than the parent quiescent galaxy population. 
Non-compact quiescent galaxies with the same stellar masses as their compact counterparts populate 
a similar distribution of environments. Thus the massive nature of quiescent compacts accounts for 
the environment dependence and appears fundamental to their history. 

Subject headings: galaxies: evolution; galaxies: formation; galaxies: fundamental parameters; galaxies: 
statistics; galaxies: stellar content; galaxies: structure 


1. INTRODUCTION 

Environment plays a significant role in galaxy mass as¬ 
sembly. Internal galaxy properties, luminosity and mor¬ 
phology, are correlated with the local galaxy density at 
z ~ 0 (e.g., Davis & Geller 1976; Park et al. 1994, 2007). 
This correlation extends to z ~ 1 and to the relation be¬ 
tween stellar mass and environment (e.g., Darvish et al. 
2015, and references therein). 

Massive compact quiescent galaxies are extreme probes 
for models of massive galaxy formation and evolution. 
These systems were first discovered at high redshift 
(z > 1) where massive quiescent galaxies are on aver¬ 
age several times smaller (i.e., have higher surface stel¬ 
lar mass density) than typical passive systems of similar 
stellar mass at z ^ 0 (e.g., Daddi et al. 2005; Longhetti 
et al. 2007; Trujillo et al. 2007; Toft et al. 2007; Zirm 
et al. 2007; Cimatti et al. 2008; van Dokkum et al. 2008; 
Buitrago et al. 2008; Damjanov et al. 2009, 2011; McLure 
et al. 2012; van de Sande et al. 2013; Belli et al. 2014). 
Recently discovered ultra-diffuse galaxies found in the 
Coma cluster (van Dokkum et al. 2015; Koda et al. 
2015) are at the other extreme of the surface density 
distribution. These objects are passively evolving. Milky 
Way sized galaxies with extremely low surface bright¬ 
ness (25 — 28 mag arcsec“^ in i?—band). The explo¬ 
ration of the links between both the extraordinarily dense 
and the unusually diffuse galaxies and their environments 
should provide further clues to the driving mechanisms of 
galaxy assembly. Using recent cosmological simulations. 
Stringer et al. (2015) traced the location of massive com¬ 
pact systems evolving into galaxies similar to NGC 1277, 
the first known local relic of a massive compact galaxy 
formed at z > 2 (van den Bosch et al. 2012; Trujillo 
et al. 2014). The results suggest that a large fraction 
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of massive compact systems are substructures of more 
massive groups and clusters of galaxies at 0 < z < 2; i.e, 
massive compacts preferentially occupy denser regions. 
These and other results (e.g., Shankar et al. 2014) sug¬ 
gest that, indeed, environment may have an important 
role in the formation, preservation and structural evo¬ 
lution of massive compact systems in the intermediate 
redshift regime. 

If size growth is efficient, massive compact galaxies 
constitute a significant fraction of the quiescent popula¬ 
tion only at high redshift (e.g., Huertas-Company et al. 
2015; Wellons et al. 2015a,b) where dense spectroscopic 
surveys, essential for characterization of the galaxy den¬ 
sity field (e.g., Geller & Hwang 2015), are not feasible. 
The recent identification of a substantial number of these 
systems at intermediate redshift (e.g., Damjanov et al. 
2015) makes the study of their environments over the 
last 4—5 Gyr possible. 

Estimates of the number density evolution of compact 
massive galaxies range from a dramatic change in the 
number density between z ^ 2 and the local universe 
(e.g., Trujillo et al. 2009; Taylor et al. 2010; van der 
Wei et al. 2014) to a very mild evolution (e.g., Poggianti 
et al. 2013a; Carollo et al. 2013; Damjanov et al. 2014, 
2015; Saulder et al. 2015; Tortora et al. 2015). Damjanov 
et al. (2015) use a spectroscopic sample of intermediate- 
redshift (0.2 < z < 0.8) quiescent galaxies drawn from 
the COSMOS survey to show that the number density of 
massive compact galaxies is consistent with no evolution 
in this redshift range. The abundance of intermediate- 
redshift compacts is similar to their number density at 
z > 1. 

The abundance of massive compact galaxies at inter¬ 
mediate redshifts provides an opportunity to study the 
evolution of these systems in much more detail than is 
currently possible at higher redshift. Large and dense 
spectroscopic surveys can provide substantial samples 
of compact systems along with central velocity disper¬ 
sion measurements for individual objects (e.g., Fabricant 
et al. 2013; Monna et al. 2014; Zahid et al. 2015, Zahid 
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et al. 2015 in prep.). Large redshift surveys covering the 
range 0.1 < z < 0.8 can probe volumes encompassing 
environments ranging from galaxy groups to the super¬ 
cluster - void network. A dense redshift survey (i.e., a 
> 90% complete spectroscopic survey given the selection) 
enables evaluation of the quiescent galaxy density field 
around compact quiescent systems on a variety of scales. 

Several studies have characterized the evolution of 
structure in the COSMOS field (e.g, Scoville et al. 2007, 
2013; Kovac et al. 2010) and investigated the links be¬ 
tween internal properties of COSMOS galaxies and their 
environments (e.g., Guzzo et al. 2007; Cassataet al. 2007; 
Capak et al. 2007; Ideue et al. 2009; Meneux et al. 2009; 
Bolzonella et al. 2010; George et al. 2011; Kovac et al. 
2013; Darvish et al. 2015). Here we use a new, dense 
and complete, redshift survey to explore the relation be¬ 
tween intrinsic and environmental properties of massive 
compact galaxies in COSMOS at 0.1 < z < 0.4. 

We combine a spectroscopic survey carried out with 
Hectospec on the 6.5-m MMT with publicly available 
data to construct a complete magnitude limited sam¬ 
ple of 880 intermediate redshift quiescent galaxies in the 
COSMOS field. The survey densely samples the distri¬ 
bution of massive quiescent galaxies at intermediate red¬ 
shift. 

We compare the galaxy stellar mass density distribu¬ 
tion around massive compact galaxies with the environ¬ 
ments of non-compact massive quiescent galaxies. Sec¬ 
tion 2 describes the spectroscopic survey and the selec¬ 
tion. We describe the technique we use to evaluate the 
galaxy stellar mass density field in Section 3.1 and our 
results are in Section 3.2. We explore the relationship 
between galaxy stellar mass and environment in Sec¬ 
tion 4. We discuss the results in Section 5 and con¬ 
clude in Section 6. We adopt IIa = 0.7, Om = 0.3, and 
Hq = 70 km s“^ Mpc“^ cosmology. 

2. THE DATA 

We use Hectospec (Fabricant et al. 1998, 2005) 
mounted on the 6.5-meter MMT to measure redshifts for 
galaxies drawn from the UltraVISTA catalog"*^ (Muzzin 
et al. 2013a). We apply a color selection {r — i > 0.2, 
g — r > 0.8) and a magnitude limit of r < 21.3. We 
refer to the Hectospec survey of the COSMOS field as 
hCOSMOS hereafter. 

We measure redshifts for galaxies without spectroscopy 
in the existing public databases (zCOSMOS redshift sur¬ 
vey, Lilly et al. 2007; Knobel et al. 2012; SDSS DR12, 
Alam et al. 2015). Our goal is to increase the redshift 
survey density in the redshift range 0.1<z<0.4asa 
basis for investigating the environments of massive qui¬ 
escent compact galaxies. 

With the inclusion of Hectospec spectroscopy, we con¬ 
struct a 90% complete sample of 1766 galaxies with 
{r — i > 0.2, g — r > 0.8) and r < 20.8. This sample 
is larger and more uniform than samples that have been 
considered previously at 0.1 < z < 0.4 (e.g.. Malt by 
et al. 2010; Huertas-Company et al. 2013b; Lani et al. 
2013; Kelkar et al. 2015). 

To define observed color cuts which provide a complete 
magnitude limited sample of intermediate-redshift qui- 

^http://www.strw.leidenuniv.nl/galaxyevolution/ 
ULTRAVISTA/Ultravista/K-selected.html 



Figure 1. Color selection of the hCOSMOS+zCOSMOS sur¬ 
vey: two upper panels show the observed g — r and r — i colors 
of SHELS galaxies (Geller et al. 2014) as functions of redshift. 
Dashed lines label our color selection: g — r > 0.8 and r — i > 0.2. 
Individual points are color-coded by the indicator of galaxy qui¬ 
escence, Dn4000. The third panel shows the fraction of quies¬ 
cent (Dn4000 > 1.44) SHELS systems with gri colors of hCOS- 
MOS-l-zCOSMOS galaxies. 

escent galaxies we examine the Smithsonian Hectospec 
Lensing Survey (SHELS; Geller et al. 2014). SHELS is 
complete to i? = 20.6 with no color selection. 

Eigure 1 shows the effect that our color selection has 
on the SHELS data for the limiting magnitude R = 20.6. 
The first two panels show g — r and r — i colors of SHELS 
R < 20.6 galaxies in the 0 < z < 0.7 redshift range. Sym¬ 
bols are color-coded by Dn4000 values measured from the 
SHELS Hectospec spectra. The g — r > 0.8 cut effec¬ 
tively provides a sample of quiescent system at z > 0.1; 
r — i > 0.2 color selection minimizes the contamination 
by foreground objects (at z < 0.1). The bottom panel 
of Figure 1 demonstrates that the fraction of quiescent 
galaxies (with Dn4000 > 1.44, see Section 2.3) in the 
galaxy sample with red gir colors increases steeply be¬ 
tween z = 0 and z = 0.1 and exceeds 90% at z ^ 0.15. 

The magnitude and color cuts we apply efficiently se¬ 
lect a complete magnitude limited sample of quiescent 
galaxies in the redshift range 0.1 < z < 0.4 (in a small 
fraction of the volume completeness is in the 80-90% 
range). Because of the broad color selection our sam¬ 
ple also contains some star-forming galaxies. Thus we 
apply additional selection criteria to construct a quies¬ 
cent subsample (Section 2.3). We base our analysis on 
the quiescent subset of 880 galaxies. 

2.1. Spectroscopy 
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The 1 square degree field of view of Hectospec is well 
matched to the size of the COSMOS field. For galax¬ 
ies with r < 21.3, a typical integration time of 1 h 
yields a redshift Fabricant et al. (2008). Because our 
intermediate-redshift sample is predominantly quiescent, 
redshift determination is based on the prominent absorp¬ 
tion features (e.g, Ca H-l-K, Balmer series, G-band) and 
on the position of 4000 A break. Strong emission lines 
(e.g, [Oil] and/or Ha) determine redshifts of emission 
line galaxies. We prioritize field positions around the 
center (02000 =10:00:28.6, ^2000 =+02:12:21.00) of the 
GOSMOS field. We also prioritize galaxies according to 
their apparent r—band magnitudes and use the Roll et al. 
(1998) software to optimize the fiber positions. We ob¬ 
tained 10 fields in varying conditions in February and 
April 2015. Damjanov et al. (2015, in prep) will include 
a detailed description of this dataset. 

We reduce the Hectospec data with the Mink et al. 
(2007) Hectospec pipeline and derive redshifts with 
HSRED v2.0, developed by the Telescope Data Gen- 
ter and based on the pipeline originally developed by 
Richard Cool^. Repeat observations show that the typi¬ 
cal error in the Hectospec redshift is 48 km s“^ (normal¬ 
ized by (1 + z); Geller et al. 2014). 

The pipeline returns the r-value (Tonry & Davis 1979) 
as a measure of the quality of the redshift. We also visu¬ 
ally inspect all of the spectra; we use only those spectra 
which yield high-quality redshifts (i.e., redshifts based on 
several reliable spectral features and with r-values + 3; 
see Fabricant et al. 2005). 

We acquired a total of 2096 new redshifts. Here we 
use 1160 of these spectra for galaxies with r < 20.8 (see 
Section 2.2) and combine them with other publicly avail¬ 
able spectroscopic data in the GOSMOS field. Table 1 
lists the source and the numbers of redshifts we use. 
The entire compilation contains 10, 314 galaxies in the 
redshift range 0 < z < 1.35 with secure spectroscopic 
redshifts, reliable photometry and GIM2D-based galaxy 
size measurements from Sargent et al. (2007)®. We use 
a subsample with high spectroscopic completeness (Sec¬ 
tion 2.2) in the central region of the GOSMOS field to 
select quiescent galaxies based on their rest-frame colors 
(Section 2.3) and to probe the environments of quiescent 
systems at intermediate redshift. 

2.2. Completeness 

We examine the completeness of the redshift survey as 
a function of position on the sky and as a function of 
limiting r magnitude. We focus on the central square 
degree where the hGOSMOS+zCOSMOS survey is the 
most dense (black rectangle in the lower panel of Fig¬ 
ure 2). 

We use the photometric Ultra VISTA catalog (Muzzin 
et al. 2013a) to define the spectroscopic completeness. 
The upper panel of Figure 2 shows the differential ratio 
of spectroscopic to photometric targets with r — i > 0.2 
and g — r > 0.8 as a function of r—band magnitude. The 
fraction of galaxies with a reliable spectroscopic redshift 
exceeds 90% for r < 20.8. To this limit, the sample 
includes 1766 red galaxies with 

®http://wTO.mmto.org/book/export/html/55 
®http://irsa.ipac.caltech.edu/data/CDSMOS/tables/ 
morphology/cosmos_morph_zurich_l. 0. tbl 
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Figure 2. Completeness of the hCOSMOS-|-zCOSMOS redshift 
survey. The upper panel shows the differential completeness as a 
function of i—band magnitude (blue). The vertical red line at r = 
20.8 marks the 90% differential completeness limit. The lower panel 
shows the spectroscopic completeness in 6' X & bins for galaxies 
with r < 20.8. The black rectangle outlines the central 1 square 
degree of the field where the differential spectroscopic completeness 
is > 90% for quiescent galaxies with r < 20.8. Black points indicate 
targets in the photometric ULTRAVISTA sample (Muzzin et al. 
2013a) without a measured redshift. 


• a spectroscopic redshift O.I < z < 0.8; 


• stellar mass 10® Mq < M* ^ 6 x 10^^ Mq (Muzzin 
et al. 2013a); 


• circularized effective (or half-light) radius 
0.2kpc < Re^c < 33kpc (Sargent et al. 2007). 


Redshifts for a majority of these objects (65%, see Ta¬ 
ble 1) are from the hGOSMOS survey. Measured radii for 
all objects exceed the average width of the Point Spread 
Function (PSF) reported for the HST AGS mosaic of the 
GOSMOS field (07095, Koekemoer et al. 2007). Muzzin 
et al. (2013a) determine stellar population parameters 
by fitting galaxy spectral energy distributions (SEDs) 
using the FAST code (Kriek et al. 2009). We use stellar 
masses based on the best-fit Bruzual & Chariot (2003) 
models with solar metallicity, a Chabrier (2003) Initial 
Mass Function, Calzetti et al. (2000) dust extinction law, 
and exponentially declining star formation histories. 

The majority of (r —i > 0.2, g — r> 0.8) color-selected 
galaxies with r < 20.8 and without spectroscopic (hCOS- 
MOS, zCOSMOS, or SDSS) redshifts are distributed to¬ 
wards the edges of the COSMOS field (lower panel of 
Figure 2). In the central 1 sq. degree of the field (black 
rectangle) there are 135 photometric targets without a 
spectroscopic redshift (< 0.8% of the spectroscopic sam¬ 
ple). A majority of the unobserved photometric targets 
(^ 60%) have r > 20.4, i.e., their magnitude distribu¬ 
tion is skewed toward the limiting magnitude. The high 
completeness results from the hCOSMOS survey design. 


2.3. The Quiescent Galaxy Sample 


Completeness [%] 
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Table 1 

Properties of the hCOSMOS+zCOSMOS redshift survey 


Sample^ 

(1) 

TotaD 

(2) 

Number of spectroscopic 
r < 20.8^ 

(3) 

sources 

Quiescent‘S 
(r < 20.8) 

(4) 

Compact® 

(r < 20.8) 

(5) 

hCOSMOS 

2096 

1160 

565 

167 

zCOSMOS 

16529 

529 

254 

81 

SDSS 

960 

77 

61 

23 

Total 

19585 

1766 

880 

271 


References: Damjanov et al. 2015 (in prep, hCOSMOS), Lilly et al. (2007, zCOSMOS), Knobel et al. (2012, zCOSMOS), 
Alam et al. (2015, SDSS DR12) 

Number of unique high-quality spectroscopic redshifts in the sample 

Number of (r — i > 0.2, g — r > 0.8) spectroscopic targets with: 1) apparent magnitude r < 20.8, 2) reliable redshift, 
3) available stellar masses, and 4) measured structural properties in the central 1 sq. degree of the COSMOS field (where 
spectroscopic completeness is > 90% to the magnitude limit, see Section 2.2.) 

Number of (r — i > 0.2, g — r > 0.8, r < 20.8) spectroscopic targets with: 1) reliable redshift, 2) available stellar masses, 3) 
measured structural properties in the central 1 sq. degree of the COSMOS field, and 4) rest-frame colors of quiescent galaxy 
population (see Section 2.3.) 

® Number of (r — i > 0.2, g — r > 0.8, r < 20.8) spectroscopic targets with: 1) reliable redshift, 2) available stellar masses, 
3) measured structural properties in the central 1 sq. degree of the COSMOS held, 4) rest-frame colors of quiescent galaxy 
population, and 5) compact structure dehned by Equation 2 
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Figure 3. Left panel: Rest-frame UVJ color-color diagram for hCOSMOS and SDSS galaxies with measured Dn4000. Color-coding 
indicates Dn4000. The separation between quiescent and star-forming galaxies (solid black line) is from Williams et al. (2009, equa¬ 
tions la andlb). Right panel: The distribution of Dn4000 for quiescent and star-forming galaxies selected based on their rest-frame UV J 
colors. 


The broad color selection of the hCOSMOS survey 
provides a complete sample of quiescent galaxies in the 
redshift range 0.1 < 2 ; < 0.4. However, at higher red- 
shifts probed by the survey the observed {r — i > 0.2, 
g — r > 0.8) color limits select a broader range of rest- 
frame colors that extends to the blue. Thus the fraction 
of star-forming galaxies increases with redshift. We ex¬ 
amine the galaxy stellar mass density field around quies¬ 
cent galaxies using these quiescent systems as tracers of 
the total stellar mass density in quiescent objects. The 


first step in our analysis is the extraction of a complete 
sample of quiescent galaxies. 

Combinations of rest-frame colors provide diagnostics 
for galaxy star-format ion activity (e.g., Williams et al. 
2009; Ilbert et al. 2010). We use rest-frame U — V and 
V — J colors from Muzzin et al. (2013a)^ to separate 
star-forming and passively evolving galaxies based on the 
selection criteria for 0 < z < 0.5 quiescent galaxies from 

"^http: //www. strw. leidenuniv.nl/galaxyevolution/ 
ULTRAVISTA/Ultravista/K-selected.html 
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Williams et al. (2009): 

U-V>1.3 (la) 

C/- E > (C - J) X 0.88 + 0.69. (lb) 

This color selected quiescent sample includes 880 
intermediate-redshift COSMOS galaxies. 

All quiescent galaxy selection criteria produce sam¬ 
ples with some contamination from star-forming outliers 
(Moresco et al. 2013). The hCOSMOS and SDSS spec¬ 
tra span the range ~ 3700 — 9000 A. Thus we can use 
Dn4000 (as defined in Balogh et al. 1999) to test the 
contamination of the rest-frame color selected quiescent 
sample. 

The Dn4000 spectral index is an indicator of galaxy 
quiescence: Dn4000 = 1.44 effectively separates 

absorption-line from the emission-line systems (Freed¬ 
man Woods et al. 2010). We follow Fabricant et al. 
(2008) procedure to measure Dn4000 for the hCOS¬ 
MOS galaxy sample and combine it with measurements 
available for the SDSS/COSMOS spectroscopic targets 
(Kauffmann et al. 2003). Fabricant et al. (2008) demon¬ 
strate that Dn4000 measured from the Hectospec galaxy 
spectra agree remarkably well with the values based on 
SDSS spectra. We compile Dn4000 measurements with 
small errors (< 0.09 of the index value, or < 2x typical 
error; Fabricant et al. 2008) for 57% of the COSMOS 
dataset. We use this subsample to examine the contami¬ 
nation of the galaxy sample drawn from their rest-frame 
colors. 

Figure 3 shows the position of hCOSMOS and 
SDSS/COSMOS galaxies as a function of their rest-frame 
{U — V, V — J) colors (left panel) and the distribution 
of Dn4000 for the star-forming and quiescent subsam¬ 
ples defined by the rest-frame UVJ selection (Williams 
et al. 2009; Equations la and lb; right panel). In the 
passive color-selected subsample ~ 15% of galaxies have 
D„4000 < 1.44; ^ 10% of galaxies with rest-frame col¬ 
ors indicating star formation have Dn4000 > 1.44. Of 
course some galaxies with large Dn4000 are star-forming 
and others with Dn4000 < 1.44 are quiescent (Freedman 
Woods et al. 2010). Thus this comparison, like all other 
classifiers, is only indicative of the possible contamina¬ 
tion. 

Moresco et al. (2013) select a set of quiescent galaxy 
samples at z < 0.5 based on a variety of selection criteria 
ranging from early-type morphologies to a combination 
of photometric and spectroscopic properties. The au¬ 
thors demonstrate that in all quiescent samples 21 — 55% 
of galaxies have prominent emission lines (the only spec¬ 
troscopic indicator of star formation used in their study). 
The contamination level of our rest-frame UV J color se¬ 
lected sample of 15% (estimated from the distribution of 
Dn4000 values) is lower than in any of the Moresco et al. 
(2013) samples tested with spectroscopic indicators. 

2.4. The Compact Quiescent Galaxy Sample 

There are a variety of definitions for compact galaxies 
(e.g., Trujillo et al. 2009; Cassata et al. 2013; van der Wei 
et al. 2014; Barro et al. 2013). In Damjanov et al. (2015) 
we show that global physical parameters like the depen¬ 
dence of the space density on redshift are insensitive to 
the compact galaxy definition. Thus we use a definition 
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Figure 4. Circularized effective radius as a function of stellar 
mass for the COSMOS compact galaxies (circles color-coded by 
redshift). The gray two-dimensional histogram represents the dis¬ 
tribution of the parent quiescent intermediate-redshift sample. The 
red dashed line shows the compactness cutoff. 

that strikes a balance between the selection of the dens¬ 
est massive systems and the necessity of having a large 
enough sample for a robust analysis of the environments. 
We use the pseudo-stellar mass surface density Ei.s: 


Si,5 = log 


f m4Mq] \ 
y(i?e.c[kpc])^-^y ’ 


( 2 ) 


and a compact galaxy threshold Si .5 ^ 10.3 (e.g., Barro 
et al. 2013; Poggianti et al. 2013a; Damjanov et al. 2015). 
Figure 4 illustrates the position of the selected compact 
systems (circles colored by redshift) in the size-stellar 
mass parameter space compared with the distribution 
of the parent quiescent COSMOS sample (gray two- 
dimensional histogram). The threshold based on pseudo- 
stellar mass surface density (Eq. 2, red dashed line in Fig¬ 
ure 4) produces a sample of 271 massive (M* > 10^° Mq) 
compact red galaxies at 0 . 1 <z< 0 . 6 . 

Figure 5 shows a cone diagram projected along the 
R.A .2000 direction. Galaxies are color-coded based on 
their pseudo-stellar mass surface density Ei.s (Eq. 2). 
Red circles represent massive compact galaxies and blue 
circles show other quiescent galaxies. One clear example 
of massive compact quiescent systems in a dense region 
is the set of nine compact systems in the core of a galaxy 
system (note the extended finger in redshift space) at z = 
0.22. Furthermore, the large number of compact systems 
at z ~ 0.35 coincides with a known significant galaxy 
overdensity in COSMOS (e.g. Masters et al. 2011). 

The bottom panels of Figure 5 cover the 0.4 ^ z < 0.6 
redshift interval and thus the largest fraction (~70%) of 
the volume. However, this redshift range contains a small 
fraction of the quiescent galaxy sample (146 objects or 
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Figure 5. Cone diagram for quiescent galaxies in the r < 20.8 hCOSMOS+zCOSMOS survey projected in R.A. 2000 - dots indicate 
massive compact quiescent galaxies (Barro et al. 2013); blue dots indicate other quiescent objects. 
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Figure 6. Absolute 1 —band magnitude vs. redshift (left) and 
stellar mass vs. redshift (right). The blue points show parent 
spectroscopic sample of quiescent galaxies. Red curves and squares 
denote the redshift evolution of the absolute magnitude limit (left) 
and galaxy stellar mass limit (right) corresponding to the magni¬ 
tude limit of the survey {rn^ = 20.8). 

^17%). Comparison with the number density of massive 
quiescent COSMOS galaxy sample from Damjanov et al. 
2015 confirms that the hCOSMOS+zCSOMOS galaxy 
sample misses > 50% of massive (M* > 10^° Mq) qui¬ 
escent systems in 0.4 < z < 0.6 redshift range because 
of the magnitude limit. Thus we limit the analysis to 
the 0.1 < z < 0.4 redshift range where the high density 
of hCOSMOS+zCOSMOS red galaxies provides a robust 
set of tracers for the quiescent galaxy stellar mass field. 

3. SMOOTHED GALAXY STELLAR MASS DENSITY 
ESTIMATION 

3.1. The Method 

We follow a standard approach for evaluating the envi¬ 
ronments of compact galaxies (Park et al. 1994; Grogin 
& Geller 1998; Muldrew et al. 2012). In particular we 
specialize to the procedures outlined by Tempel et al. 
( 2012 ). 

To evaluate the galaxy stellar mass density field 
(GSMD) around quiescent galaxies we begin by con¬ 
structing a continuous GSMD field. We identify 880 
quiescent galaxies covering stellar mass range between 
10® Mq and 6.3 x 10^^ Mq. We adopt a correction fac¬ 
tor (see e.g., Grogin & Geller 1998; Tempel et al. 2012) 
to account for the unsampled end of the mass function. 
We transform the corrected point distribution of galaxy 
stellar masses in our magnitude-limited sample into a 
continuous GSMD field. 

In a magnitude limited sample, the fraction of galax¬ 
ies more luminous than some fixed minimum luminosity 
naturally increases with redshift. Here we are interested 
in the GSMD rather than the luminosity density. Be¬ 
cause we select quiescent galaxies, translating from one 
limit to the other is a reasonable approximation. 

We assume that the galaxy stellar mass function is in¬ 
dependent of compactness. We use the galaxy stellar 
mass function from Muzzin et al. (2013b); this galaxy 
stellar mass function is derived for the redshift range 
0.2 < z < 0.5 of the GOSMOS survey. Thus assuming 
this form is internally consistent. 

We use the galaxy stellar mass function to assign a 
redshift-dependent weight to each galaxy, Wz , that com¬ 
pensates for the incompleteness: 


^ /^"M,,4>(M0dM. 

where $ (M*) is: 


( 3 ) 


$(M) = (In 10) 


2q(M—M*)( l-|-a) 


xexp 


_^q(M-M*) 


( 4 ) 

Here M = log(M*/MQ) is galaxy stellar mass, a = 0.92 
is the slope at low mass, M* = log{M* /Mq) = 11.25 is 
the characteristic mass, and is the normalization of 
the stellar mass function (irrelevant here). This galaxy 
stellar mass function is based on galaxy stellar masses 
from the COSMOS/UltraVISTA catalog (Muzzin et al. 
2013a, Section 2). 

For the lower and upper stellar mass cuts we use 
the limiting galaxy masses we sample (see Section 4): 
Ml = log{Ml/MQ) = 9 and M„ = log(M“/MQ) = 11.8. 
Because of the approximate constancy of the mass-to- 
light ratio (M/L)^ with absolute magnitude for quiescent 
galaxies with Dn4000 >1.4 (e.g., Geller et al. 2014) the 
effective galaxy stellar mass limit at the galaxy redshift 
is directly related to the absolute magnitude 
limit Mr^Um{z). 

We determine the absolute r—band magnitude at the 
observed redshift by synthesizing photometry in this 
band from the measured multi-band UV-IR spectral en¬ 
ergy distribution (SED). We determine the Tf—corrected 
and reddening corrected magnitude by fitting stellar pop¬ 
ulation synthesis models of Bruzual & Chariot (2003) to 
the observed SED using the LePHARE code (Arnouts 
et al. 1999; Ilbert et al. 2006). The absolute magnitude 
limit (red solid line in the left panel of Figure 6) is 


M^r,lim{z) — eilr^lim 5 log ^ C ^ ^ ^ 

where mr^um = 20.8 is the apparent magnitude limit, 

Dl{z) is the luminosity distance, and K(z) is the median 
AT—correction. 

At each redshift we translate the magnitude limit 
Mr,Um{z) into the galaxy stellar mass Mumiz) (red 
solid line in the right panel of Figure 6) using 
(M/L)t ~ 1, based on the approximately constant 
{M/L)r value for systems with Dn4000 > 1.4 in our 
hCOSMOS-bzCOSMOS quiescent sample. Only 32 
galaxies (3.6% of the sample) have stellar masses below 
this galaxy stellar mass limit. 

Values of the redshift-dependent weighting factor 
are not large: they range between 1.00 and 1.16 for galax¬ 
ies in the 0.1 < z < 0.4 redshift interval. The esti¬ 
mated effective total stellar mass per galaxy from the 
magnitude-limited sample is then 

Mf ‘ = M^xWz. (6) 

The weighted stellar masses are the basis for the 
smoothed GSMD field. 

We smooth the region around each galaxy in the sam¬ 
ple with a i ?3 spline kernel (Tempel et al. 2012): 










Damjanov et al. 


\x -2\^ - A\x - Ip + 6|a;p - A\x + Ip + |a; + 2|' 
B,{x) = - - - 

. 

where x = \r — ri\/a is the distance between a given 
point r = [a, (5, r{z)] and the i—th galaxy, normalized by 
the smoothing scale a. The GSMD held at each position 
is a sum over all galaxies in the sample: 


m = 


N 




2=0 



Mi 


( 8 ) 


where i^adr — ril/a) = 0 for |r —^ 2a. The number of 
positions where we calculate the GSMD held is equal to 
the number of galaxies in the sample. In order to avoid 
overweighting each galaxy with the spline, we sample the 
effective GSMD held at randomly chosen points around 
each object. We require that both the offsets from these 
random positions and the hducial galaxy lie within the 
same cube of side length 1 Mpc. In effect, we calcu¬ 
late the GSMD held in non-empty cells of a cartesian 
grid. This analysis produces a continuous distribution of 
GSMD that we use to compare the GSMD held around 
massive compact systems with the GSMD held around 
the underlying quiescent population. 


3.2. The Galaxy Stellar Mass Density Field Around 
Compact Quiescent Galaxies 

We calculate the GSMD held for two smoothing 
lengths, 2 Mpc and 3 Mpc. The smoothing scales we 
explore are limited by: 1) the density of our targets in 
redshift space, and 2) the transverse dimension of the 1 
sq. degree held at the limits of the redshift range we 
probe. 

As discussed in Section 2.4, the sampling of redshift 
space for z > 0.4 is sparse. Environment determinations 
on the scales we can explore at lower redshift become 
dominated by shot noise because the mean galaxy sepa¬ 
ration, 8—10 Mpc at z > 0.4, substantially exceeds even 
the 3 Mpc smoothing length (Grogin & Geller 1998). 

The mean galaxy separation at the peak of the redshift 
distribution (z ~ 0.34) is ~ 5 Mpc. Simulations show 
that the reconstruction of over-dense regions fails for 
smoothing lengths much smaller than the mean galaxy 
separation in a densely populated redshift bin (Gucciati 
et al. 2006). Thus we select 2 Mpc as the smallest 
smoothing scale for environment density estimation. At 
the scale of 2 Mpc ~ 75% ( 542/734) of the bins contain 
more than one galaxy. 

The largest smoothing scale we probe is set by the 
transverse dimension of the field at the lower redshift 
limit of our sample. The central 1 sq. degree of the 
GOSMOS held covers a scale > 6 Mpc for z ~ 0.1. The 
scales covered by the survey must be substantially larger 
than the smoothing length to avoid edge effects. Thus 
we select 3 Mpc as the upper cut-off for the GSMD held 
smoothing scale; it is the largest scale where we can use 
the full extent of the redshift range. 

The GSMD held near the edge of the survey is underes¬ 
timated if the smoothing length covers a region outside 
the survey area (e.g., Tempel et al. 2012). To account 
for the edge effects we estimate the GSMD only in sec¬ 
tions of the survey area that are farther than a smoothing 



Figure 7. Distribution of the GSMD around galaxies in the parent 
quiescent sample (blue) and for the quiescent compact subsample 
(red) sampled on a scale of 2 Mpc (left column) and 3 Mpc (right 
column). Top histograms show normalized probability densities. 
Central panels show cumulative distributions with the correspond¬ 
ing p— values of the two-sample Anderson-Darling (A-D) test. The 
number distribution of galaxies in bins of top histograms are in the 
bottom panels. For both scales the two density distributions are 
inconsistent with being drawn from the same parent population, 
as indicated by the small p—values. 

length from survey edges. For a given smoothing scale, 
the fraction of the survey volume excluded from the den¬ 
sity estimation is a strong function of redshift. At z ~ 0.1 
and for a = 3 Mpc we include only the central ~ 10% 
of the survey. This fraction rises to ^ 55% at z ^ 0.2 
for the same smoothing length. By considering only ob¬ 
jects far enough from the survey boundaries we obtain 
an unbiased estimation of the GSMD field. 

Figure 7 shows the distribution of the GSMD field 
around quiescent GOSMOS galaxies at 0.1 < z < 0.4 
(blue histograms) compared with the GSMD field sur¬ 
rounding massive compact systems (red histograms). 
The distribution of the GSMD field around massive com¬ 
pact galaxies shifts toward the high-density end of the 
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Figure 8. Top panels: GSMD map for the smoothing length a = 2 Mpc in three redshift slices encompassing similar volumes. Gray circles 
denote the positions of compact quiescent galaxies. Bottom panels: Spatial distributions on the sky of the parent quiescent sample (blue 
circles) and the massive compact subsample (red circles). It is visually evident that the compact galaxies are generally in denser regions. 


distribution for the parent quiescent sample (top panels 
of Figure 7). This shift holds for both smoothing lengths. 

The two-sample Anderson-Darling (A-D) test (Scholz 
& Stephens 1987), based on the cumulative distributions 
in the two central panels of Figure 7, corroborates the 
visually apparent difference between the two sets of his¬ 
tograms. Difference between cumulative distributions is 
much larger for a = 2 Mpc smoothing length. The proba¬ 
bility that the two distributions are drawn from the same 
underlying distribution is p = 8 . x 10“® for the 2 Mpc 
scale. Thus for the smaller smoothing length the null 
hypothesis is rejected at a 4.3 ct significance level. The 
probability is lower for 3 Mpc scale: p = 1.27 x 10“^ 
gives a marginal 2.24 cr rejection. This lower significance 
of the 3 Mpc result reflects the limitations of our 1 sq. 
degree magnitude-limited (r < 20 . 8 ) survey. 

The number of quiescent and compact quiescent galax¬ 
ies in the majority of the histogram bins in Figure 7 ex¬ 
ceeds n = 10. Numbers are lower only for the two most 
extreme-density bins (bottom panels of Figure 7). At 
2 Mpc the resulting contrast between the GSMD dis¬ 
tributions for the compact and for the parent quiescent 
samples is thus not affected by small-number statistics. 

The top panels of Figure 8 show maps of the GSMD 
field, constructed using a smoothing scale of 2 Mpc, in 
three redshift slices (of similar volumes) between z = 0.2 


and 2 = 0.4. We derive these density maps based on 
the spatial distribution of quiescent galaxies (blue circles 
in the bottom panels of Figure 8 ). To construct GSMD 
maps we use a linear radial basis function to interpolate 
between the positions where we evaluate the GSMD. We 
avoid the edges of the survey (areas outside dashed-line 
squares in Figure 8 ) because the GSMD is underesti¬ 
mated in regions where our sample is not complete. In 
each redshift slice, massive compact galaxies (gray circles 
in the top panels and red circles in the bottom panels of 
Figure 8 ) favor denser regions (see also Figure 7). 

We also examine the GSMD distribution for mas¬ 
sive compact quiescent galaxy samples selected based on 
more conservative compactness thresholds. All compact 
samples exhibit the same preference for denser regions. 
However, a smaller number of systems in the samples 
with more extreme properties decreases the significance 
of the two-sample A-D test results. For the compact sam¬ 
ple of 50 quiescent galaxies with £ 1,5 > 10.5 (Eq. 2), the 
null hypothesis that the GSMD distributions for the com¬ 
pact and for the parent quiescent sample originate from 
the same underlying distribution is rejected at a marginal 
2.3tr significance level {p = 1.07 x 10“^). The sample of 
massive quiescent compact systems with Si .5 > 10.7 con¬ 
tains only 10 objects; obviously inadequate for statistical 
analysis. A study of the environments of intermediate- 
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Figure 9. Distributions of galaxy stellar masses for 1) the par¬ 
ent quiescent galaxy sample (blue) and 2) the compact subsample 
(red). The first panel show normalized probability densities. The 
second panel shows two cumulative distributions. The distributions 
of absolute galaxy numbers are in the third panel. The compact 
quiescent galaxies occupy the high-mass end of the parent stellar 
mass distribution. 

redshift galaxy samples covering a broad range of com¬ 
pactness levels requires a survey covering a field larger 
than the COSMOS field. 

4. INTERPRETATION: THE IMPACT OF GALAXY 
STELLAR MASS 


The denser environments of massive compact galaxies 
may be related to their stellar mass and/or to their dense 
internal structure (i.e, a combination of their mass and 
size). Here we explore the impact of the massive nature 
of compact galaxies. We use magnitude-limited samples 
of non-compact quiescent galaxies and compact quiescent 
galaxies with the same stellar mass distribution to test 
whether the distribution of the GSMD around compact 
and non-compact objects differ. 

At z < 1 massive (M* > 5 x 10^° M*) red galaxies, 
regardless of compactness, reside preferentially in high- 
density regions (e.g., Bolzonella et al. 2010; Darvish et al. 
2015 (COSMOS), Mortlock et al. 2014 (CANDELS - 
UDS and GOODS-S)). Thus the difference in the stel¬ 
lar mass distribution of our parent quiescent sample and 
the compact subsample may account for the observed 
difference in environments (Figure 6). 

Massive compact galaxies populate the higher-mass 
end of the distribution for the underlying quiescent pop¬ 
ulation (Figure 9). The parent quiescent galaxy sam¬ 
ple spans the galaxy stellar mass range from 10® Mq to 

6.3 X 10^^ Mq, with a median value of M* = 6 x 10^® Mq. 
The minimum stellar mass for the compact subsample is 
M* = 10^® Mq, and the median mass is M* = 10^^ Mq. 
The two stellar mass distributions differ: a p— value of 
3.39 X 10“® corresponds to a 6.2 cr rejection of the hy¬ 
pothesis that the two stellar mass distribution originate 
from the same parent sample (central panel of Figure 9). 

To further investigate the relationship between the 
high stellar mass of compact galaxies and their surround¬ 
ing GSMD, we use the compact stellar mass probabil¬ 
ity distribution (red hatched area in the top panel of 
Figure 9) to select a sample of non-compact (Ei ,5 < 

10.3 log(MQ kpc“^'^), see Eq. 2) quiescent objects with 
the same stellar mass distribution as the compact sample. 
We then evaluate the GSMD field around the most mas¬ 
sive non-compact quiescent systems using the smooth¬ 
ing kernel density estimation technique described in Sec¬ 
tion 3.1. 

In Figure 10 we compare the GSMD distribution 
around massive compact (red histogram) and non¬ 
compact objects (green histogram) for the 2 Mpc 
smoothing length (see Section 3.2). The distributions for 
the two types of objects are indistinguishable {p ^ 0.5, 
central panel of Figure 10). This comparison suggests 
that the observed tendency of massive compact galaxies 
at 0.1 < z < 0.4 in the GOSMOS sample towards re¬ 
gions of greater surrounding GSMD is mainly driven by 
the greater stellar masses of compact systems. 

If the environment were strongly related to compact¬ 
ness rather than or in addition to the mass, we might ex¬ 
pect a residual difference between the distributions pre¬ 
sented in three panels of Figure 10. Much larger samples 
might enable detection of a difference between similarly 
constructed distributions. 

5. COMPARISON WITH PREVIOUS WORK 

Several recent studies investigate the relationship be¬ 
tween massive (M* > 10^® Mq) galaxy size and environ¬ 
ment over cosmic time, with diverse results. Some of 
the results at high redshift (z > 1) suggest that quies¬ 
cent color-selected galaxies in clusters on average have 
larger sizes than their massive analogs in less dense re- 
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Figure 10. Distributions of GSMD at a 2 Mpc smoothing scale 
for: 1) a randomly selected subsample of non-compact quiescent 
galaxies with the same stellar mass distribution as the compact 
galaxy subsample (green), and 2) the quiescent compact subsam¬ 
ple (red, as in Figure 6). The top histograms show the normalized 
probability density; the corresponding numbers of galaxies are in 
the bottom histograms. The central panel shows cumulative den¬ 
sity distribution for the two subsamples. The GSMD distribution 
for the massive non-compact subsample is indistinguishable from 
the equivalent distribution for compact galaxies. 


gions (Cooper et al. 2012; Papovich et al. 2012; Delaye 
et al. 2014). A similar difference apparently occurs be¬ 
tween quiescent galaxy sizes in the highest and the lowest 
density regions (Lani et al. 2013). However, other high- 
redshift studies suggest the absence of any trend with 
environment (van der Wei 2008; Rettura et al. 2010; New¬ 
man et al. 2014). Still others find that quiescent galaxies 
with early-type morphologies appear more compact in 
clusters than in lower density regions (Strazzullo et al. 
2010; Raichoor et al. 2012). 

Depending on the sample selection and definition of the 
environment, previous analyses of intermediate-redshift 
quiescent galaxy samples paint an equally confusing pic¬ 
ture. Both galaxy counts in a fixed physical aperture 
and the distance to the nearest n-th neighbor suggest 
that at 0.5 < z < 1 quiescent galaxies are larger in high 
local galaxy density environments. This positive trend 
in size with local galaxy density is less pronounced than 
at z > 1 (Lani et al. 2013). Group galaxies in the COS¬ 
MOS field at 0.2 < z < 1 selected either based on red 
rest-frame colors or on early-type morphology follow the 
same size-stellar mass relation as their counterparts in 
less dense regions (Huertas-Company et al. 2013b). Sizes 
of massive early-type galaxies in the EDisCS survey at 
0.4 < z < 0.8 appear to be independent of the environ¬ 
ment (Kelkar et al. 2015). The authors find no trend 
in size with environment for a quiescent intermediate- 
redshift sample selected based on red rest-frame B — V 
color. 

At z ^ 0 the results are also inconclusive. Early- 
type galaxies appear to follow the same size-mass rela¬ 
tion in all environments (Maltby et al. 2010; Huertas- 
Company et al. 2013a). However, cluster galaxies in the 
PM2GC and WINGS survey display a different trend: 
quiescent dense galaxies are proportionally more nu¬ 
merous in clusters than in other types of environments 
(Valentinuzzi et al. 2010; Poggianti et al. 2013b). A large 
z < 0.12 dataset drawn from SDSS DR7 shows that mas¬ 
sive (M* > 4 X 10^° Mq) early-type galaxies are smaller 
in the over-dense than in under-dense regions (Cebrian 
& Trujillo 2014), although the difference in size is not as 
pronounced as in PM2GC- and WlNGS-based studies. 

To understand the range of results it is important to 
distinguish the impact of size from the impact of mass. 
In order to make this distinction, samples must be se¬ 
lected with identical mass distributions (Section 4) . The 
well-known correlation between galaxy intrinsic luminos¬ 
ity and/or stellar mass and local density requires this 
approach (Park et al. 2007; Scodeggio et al. 2009; Bol- 
zonella et al. 2010). It is unclear whether other studies 
take this known dependence into account. 

Once the mass is taken into account, the variety of 
results may persist because of 1) the use of photometric 
redshifts, 2) small, incomplete galaxy samples (especially 
at high redshift), 3) the range of environments probed, 
4) and the techniques used for GSMD field estimation. 
Thus a direct quantitative comparison among these stud¬ 
ies is not possible. 

Other observational results show a trend toward 
smaller sizes of z < 0.12 massive early-type galaxies in 
overdense regions in SDSS (Cebrian & Trujillo 2014) and 
in nearby clusters (Valentinuzzi et al. 2010; Poggianti 
et al. 2013b). Wellons et al. (2015b) indicate that their 
theoretical results are also consistent with Valentinuzzi 
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et al. (2010). Results presented here may be consistent 
with the observed trends in low-redshift clusters (and 
thus in agreement with some theoretical predictions of 
Wellons et al. 2015b), but our sample is not large enough 
to discriminate between the impact of mass and compact¬ 
ness. 

A concern in comparing the observations with theory is 
that the simulations (e.g., Wellons et al. 2015a) predict 
a steep decline in the number density of massive com¬ 
pact quiescent systems with redshift, discrepant with the 
most recent observational evidence (e.g., Poggianti et al. 
2013a; Carollo et al. 2013; Damjanov et al. 2014, 2015; 
Saulder et al. 2015; Tortora et al. 2015). It is noteworthy 
that the observations indicating a much milder decline 
in the abundance of massive compact quiescent systems 
cover large volumes (e.g., Damjanov et al. 2014; Saulder 
et al. 2015; Tortora et al. 2015), much larger than the 
simulations (Wellons et al. 2015a). Resolution of these 
issues and a more extensive consideration of the relations 
among galaxy compactness, mass, and environment are 
an impetus for surveys covering larger volumes than of¬ 
fered by COSMOS. 

6. CONCLUSIONS 

The abundance of massive compact galaxies at inter¬ 
mediate redshift enables investigation of the relation be¬ 
tween the internal properties of these systems and their 
environments based on a dense spectroscopic survey. We 
conduct a spectroscopic survey of the central 1 sq. degree 
region of the COSMOS field (hCOSMOS, Damjanov et 
al. 2015 in prep.) to construct 90% complete sample to 
the limiting magnitude = 20.8. 

We use the redshift survey to select a complete mag¬ 
nitude limited sample of quiescent galaxies based on the 
combination of U — V and V — J rest-frame colors. The 
Dn4000 > 1.44 spectroscopic indicator of galaxy quies¬ 
cence agrees well with the rest-frame UV J selection. 

We estimate the GSMD for two smoothing lengths (2 
and 3 Mpc) around quiescent systems at 0.1 < z < 
0.4. Using the compactness definition from Barro et al. 
(2013), we select a sample of massive compact quiescent 
systems and compare their environments with the GSMD 
field of the parent quiescent population. 

Massive quiescent compact systems prefer denser re¬ 
gions (Figure 8). At a 2 Mpc smoothing scale the GSMD 
for massive compacts is significantly offset toward higher- 
density relative to the one for the complete quiescent 
sample. The difference in the stellar mass distributions 
between compact and non-compact quiescent galaxies 
completely accounts for this shift. 

The massive nature of quiescent compact galaxies ap¬ 
pears to be fundamental to their formation and evolu¬ 
tion. There may be a further dependence of the size 
of the objects on environment, but there is no detectable 
signal of this additional effect in our sample. Much larger 
samples might enable exploration of the possibly subtle 
impact of the size. A survey over a larger area would en¬ 
able exploration of a broader range of smoothing scales. 
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